Aiming at the problem that the target regions of rectal cancer show different sizes， shapes， textures， and boundary clarity on Magnetic Resonance Imaging （MRI） images， to overcome the individual variability among patients and improve the segmentation accuracy， an Adjacent Slice Attention Fusion Network for rectal cancer segmentation （ASAF-Net） was proposed. Firstly， using High Resolution Network （HRNet） as the backbone network， the high-resolution feature representation was maintained during the feature extraction process， thereby reducing the loss of semantic information and spatial location information. Secondly， the multi-scale contextual semantic information between adjacent slices was fused and enhanced by the Adjacent Slice Attention Fusion （ASAF） module， so that the network was able to learn the spatial features between adjacent slices. Finally， in the decoder， the co-training of Fully Convolutional Network （FCN） and Atrous Spatial Pyramid Pooling （ASPP） segmentation heads was carried out， and the large differences between adjacent slices during training was reduced by adding consistency constraints between adjacent slices as an auxiliary loss. Experimental results show that compared with HRNet， ASAF-Net improves the mean Intersection over Union （IoU） and mean Dice Similarity Coefficient （DSC） by 1.68 and 1.26 percentage points， respectively， and reduces the 95% mean Hausdorff Distance （HD） by 0.91 mm. At the same time， ASAF-Net can achieve better segmentation results in both internal filling and edge prediction of multi-objective target regions in rectal cancer MRI image， and helps to improve physician efficiency in clinical auxiliary diagnosis.